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To the President of the Senate and' the i 
Speaker of tho House of Representatives 

This report discusses problems of the E.'per imental 
Schools program and lessons learned from it wh ich should be 
applied to the management of future educational research 
efforts. The program was established under authority of the 
Cooperative Research Act of 1954, as amended [20 U.S.C- 331a), 
to test the hypothesis that comprehensive chongcs to existing 
educational systems will improve the quality of education. 
The program is administered by the National Institute of Ed- 
ucation, Department of Health, Education, and Welfare. 

We made our review pursuant to the Budget and Accounting 
Act, 1921 (31 U.S.C. 53), and the Accounting and Auditinq Act 
cZ 1950 (31 U.S.C. 67). 

Vie are sending copies of this report to the Director, 
Office of Management and Budget, ^snd to the Secretary of 
Health, Education, and Welfare. 

■ - Comptroller General 

of the United States 
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COMPTKOLLER GENERAL'S 
REPORT TO THE CONGRESS 



EXPERIMENTAL SCHOOLS PROGRAM: 
OPPORTUNITIES TO IMPROVE THE 
MANAGEMENT OF AN EDUCATIONAL 
RESEARCH PROGRAM / 
National Institute of Education 
Department of Health, Education, 



and v^elfare 



D I. G E S T 

According to the National Institute of Edu-- 
cation, the Experimental Schools program 
was established to test the hypothesis that 
comprehensive changes to existing educational 
systems will improve the qual ity of educa- 
t ion. It was des igned to increase know- 
ledge about the process of .education and to 
implement the changes resulting from re- 
search, demonstration, and experimentation 
carried out in actual school settings* 

The program, established within the Office 
of Education in 1970, was transferred to the 
Institute in August 1972, when the Institute 
was established as the Feda-ral Government's 
focal point for educational research. Both 
the Institute and the Office of Education 
are part of the Education Division within 
the Department of Health, Education, and 
Welfare. (See p. 1, ) 

GAO found some improvements were needed in 
planning and in carrying .cut education re- 
search programs. 

Accordingly, GAO recommends that th»? Secre- 
tary of Health, Education, and VJelfare direct 
the Director of the Institute, wherever ap- 
pl icable, to: 

— Insure that project plans approved for 
funding include (1) documentation of the 
need for the ediicat ional changes sought 
through the rerearch, (2) specific in- 
structions for doiiiq the research, and 
(3) specific, measurv-^ble objectives in 
terms of output or impact. (See p. 26.) 



T f g r Sheet . Upon removal, the report 
rover date, should be nated hereon. 
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— looure that evaluat ion will provide 
necessary impact and cost information 
over the life of the croject. (See 

— Institute procedures for requiring pro- 
gram offices to ver ify that recipients ' 
account ing systems will produce the type 
of data necessary to insure compl ip.nce 
with special program financial regula- 
tions. (See p. 31.) 

— Institute procedures to insure that re- 
cipients give this d3ta to program of- 
fices . ( See p. 31 . ) 

The Department of Health, Educat ion, and 
Welfare agreed generally with GAO's recom- 
mendations. The Department said, however , 
that GAO's report did not inclu*;e enough 
bacUgrocnd or. 1 argc-scale social sc ience 
research and development methodology or on 
the d if f icult ies vi th such research and de- 
ve lopment , The Department also expressed 
concern about the ni:nber of projects GAO 
reviewed and the t Irving of its review. 

GAO does not wish to minimize the difficul- 
ties involved in 1 arge-scale soc ial sc ience 
research and development activities of this 
type. 

GAO's report assesses the Institute's man- 
agement of the research and development 
performed and doss not deal with program 
management m re la t ion to the state of the 
art of educational research and development. 

Rather, GAO^s purpose is to stimulate action 
by the Institute so that management problems 
discussed in this report will not recur in 
the future, 

GAO's recommendations were prompted by the 
following problems at the projects visited: 

— Projects generally had not prepared plans 
wiilch could be used to effectively carry 
out and eva luate comprehens ivo ed uca- 
tional changes. The plans were written 
in conceptual , rather than operational, 
terms. (See p, 10. ) 

ii 
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— Evaluations did not prcrluce adequate in- 
formation on projects: * impact on students , 
tea jhers , admin Is trators , and comnrrun it ies . 
{ See p. .13 . ) 

— Non:- of the pr o j ects had accumulated 
enouvih baseline data on student achieve- 
ment ar.c* dctitudinal levels cither be- 
fore the comprehensive changes were made 
or early in the proiects' operation. As 
a resultr evaluators will not be able to 
fully determine the impact of the compr e- 
hensive educational changes over the 
5-year project life. (See p. 21.) 

— At four of the five projectSr program 
evaluators had not made any cos t analyses 
at the t ime of GAO ' s visit. Accord ing 
to the Institute, understanding (1) the 
rost of an innovation and (2) the shifts 
in the traditional spend ing pat ter ns to 
accommodate the innovat i ■ are important 
to: 

— The educat ional pr act i t ioner try- 
ing to dec ide on the innovat ion ' s 
ut il i ty , adapt ion , arul implements- 
t ion. 

— The educat ional researcher try ing 
to understand the process and the 
problems of phas ing out Federal 
funding. (See p. 22. ) 

— The Experimental Schools program did not 
set out specific, measurable object Ives' 
for evaluat ing its ef f ect iveness , Also 
individual projects were not required to 
establish similar object ives , wh ich would 
have allowed for object ively measur ing 
the effect iveness of a program involving 
comprehensive changes. (See p. 23.) 

— The Experimental Schools prc^jram did not 
insure that participating school districts 
could provide the data necessary to deter- 
mine compliance with special program 
financial regulations. At three of the 
five projects, GAO found that the records 
did not include the type of data needed. 
(See ch. 3,) 
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CHAPTER 1 
INTRODUCTION 

According to the National Institute of Education (NIE) , 
the Experimental Schools program is an educational research 
effort designed to test the hypothesis that comprehensive 
changes x^o existing educational systems will improve the 
quality of education. The program is also designed to in- 
crease and to improve basic knowledge about the process of 
education and to implement the results of ■ ^search, demon- 
irtrat'ion, and experimentation in actual scii'. ol sett\ngs. 

The Office of Education, Department of Health, Education, 
and Welfare (HEW) began the. program in 1970 under authority 
of the Cooperative Research Act of 1954, as amended (20 
U.S.C. 331a). This act authorizes the Office of Education to 
make grants to (1) universities and colleges, (2) other 
public or p 'ivate agencies, institutions, and organizations., 
and (3) individuals for research, surveys, and deraonstrations 
in the field of c?ducation and for the dissemination of infor- 
mation derived from educational research. 

NIE assumed responsibility for the Experimental Schools 
program in August 1972 when NIE v/as established within HEW 
by the Education /^inenc'.iients of 1972 (20 U.S.C. 122ie (SUPP. 
ii. 1972)). Both NIE P.nd the Office of Education are part 
of the Education Divi?i<^n v/ithin HEW. 

NIE was established by the Congress to: 

•»* * * conduct educational research; collect and 
disseminate the findings of educational research; 
train individuals in educational research; assist 
and foster sucli research, collection, dissertiina- 
tion, or training through grants, or technical 
assistance to, or jointly financed cooperative 
arrangements with, public or private organiza^ 
tions, institutions, agencies, or individuals. " 

The E>:per imental Schools program has funded 18 projects, 
15 of v;hich are operated as a part of local school systems. 
Three projects are administered by local Urban Leagues 
through a grant to the National Urban League. Projects 
operated by local Urban Leagues — refe'^red to as street 
academies — are aimed at low-income and/or minority students 
who have dropped out of school or who are achieving below 
their potential in the regular school system. 

NIE*Vias stated that no other projects will be funded 
and the program will be terminated when Federal support of 
established projects ends. Each project is planned to 

1 
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operate 5 years. The last projects to ha selev^ted were 
forded in fiscal year 1973 and will be completed in fis^-al 
year 197b. 

ADM INISTRAT ION AND O PERATION 
OF TH£ pr ogram " ^ 

Three separate competitive announcements were made to 
select projects for the Experimental Schools program. 
Although the theire of Scich competition was different, the 
selection process used was the sarne. 

The first competitive announceraent , made in December 
1970, contained basic program policies and general guidelines 
for projects. Project proposals were to be designed to test 
and to demonstrate the relative effectiveness of combinations 
of various educational research products, practices, and ideas 
showing promise for improving American education- 
Each project was to be organized around a central educa- 
tional concept that reflected change from the educatic aJ 
setting that existed at the local level. to what the school 
ssystem and the local conuuunity believed education oi^ahc to 
be. 

The announcej.i-nt set out the c riteria to be used in 
selecting sites fo^ projects to begin operations in schoo] 
year 1971-72. These criteria included: 

--Demonstrated experience with educational innovations 
on^ a large scale. ' ' 

Staff capacity and competency to manage comprehensive 
experimentation. 

Development of a plan for broad participation in the 
design^ implementation, and governance of a project. 

Identification of the target population. 

— Extent to which the design fulfilled the objectives 
of the Experimental Schools program, including a: 

1- Primary target population of low- income 
children. 

2. Target population of about 2000-5000 students. 

3. Longitudinal kindergartGn~to-12th-grade design. 
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4. Comprehensive approach to the learning environ- 
ment, including, but not limited to, curriculum 
development, comjnunity participation, staff 
development, administration, and organization. 

— Attention to evaluation and documentation. 

— CommitiT:Cnt of resources for the duration of the 
project. 

The second competition was announced in March 1971. The 
announcemant solicited proposals for projects which would; 
represent signiricunt alternatives to (1) the learning 
experiences be t ng offered to students, (2) tho way those 
experiences wers structured and organized, and (3) the relation- 
ship between the educational program and the community. 

According to this announcement, proposalis vere to 
answer the following (questions: 

--How well does the proposed project fit the description 
of an Experimental Schools project as set out by the 
announcement? 

--How strong is the vidence that the comDono;)ts of the 
project are compatible, and mutually r-in n.. ^cincj? 

How strong is the evidence that the <^>.:: national 
problem(3) co be addressed is/are peri:.;i.ant to the 
target population and how well does the project 
address the problem t-O and the needs of the tirqet 
population? 

How strong is the evidence that the applicant is 
capable of carrying out the project as stated? 

1,^ \ hai, extent does the comprehensive design of the 
propo.?CG project make it a significant alternative 
to exist vtT school programs? 



The arinouncement further stated that all project 
fi^rst^^'^^^ ^^^^ ii-nplemented during each project's 



The third competi ,ion was announced in March 19 72. 
Through it, the program made available to a limited nui'ser 
of rural school systems the opport^onity to test new ideas 
tor educational improvement in and for small rural schools 
School systems located in rural settings and with up to 
2,b00 students qualifi-?d. 
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After each competition independent selection coitimittees 
made up of non-^Goverr.^rrat educational experts reviewed the 
proposals and selected 8 to 10 school systems to receive 
planning grants. School systems selected as project sites 
during the first two competitions received grants of between 
$10,000 and $45,000 to cover their planning efforts, which 
Were to be completed within 2 to 8 months* The 10 rural 
school systeros selec' 3d during the third competition 
received l-^yea^' plahi.in- grants of from $46,500 to $121,400. 
The amount of th\^ planning grants was based on the capability 
of local scho--): distiJ/^ts to plan for comprehensive change. 

Applicants receiving grants were to use the grants to 
prepare complete, detailed plans. The sel^iction coitmiittees 
reviewed -these plans and selected projects to receive opera- 
tional funding. 

Under the Office of Education, the Experimental Schools 
program was originally planned to provide operational support 
for two 30-month periods during the 5-year term of each 
project. However, under NIE^ funding support is now provided 
on a yearly basi3. Although projects are reviewed a'^ the 
f'.nd of each fund:.rig period, full 5-year funding for each 
px*oject is virtually assurer. Funding suppr -rt" for each 
project is limited to the incremental costs associated with 
implementing it, such as the costs for developing the staff 
necessary for operating the program, acquiring materials, 
doing minoj; remodeling, and evaluating and docUiii:inting the 
project. 

To receive prrogram funds, applicants were required to 
indicate a willingness to continue projects with their own 
resources after Federal support ended. In each successive 
year of the program, the local school district is to provide 
an increasing percentage of total project costs; thus, it 
gradually assumes the total cost. This requirement is 
intended to help insure that (1) changes made are not stopped 
after 5 years because of costs and (2) the ideas tested have 
the potential of being economically feasible for other school 
systems .to implement. 

FUNDING 

The Experimental Schools program has awardec^i contracts 
and grants^ totaling about $48 million from its start in 
December 1970 through June 30, 1975. NIE estimates- that an 
additional ?7 million will be required to completely fund 
all program projects through fiscal year 1978. 

We reviewed five projects funded during the first two 
conpetitions. NlE estimated that these projects and related 
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evaluation contractors will receivG S36 million of the 

$55 million to be sp^nt by the Experimental Schools program. 

PROJECT CliARACTERISTICS / 

Projects were originally designed to institute compre- 
hensiva changes in only a portion of the schools within 
participating school districts; however, they were to include 
all grade levels — kindargarten through the 12th grade.' 
Proi^acts are comprised of several compcr.^nts — for example, 
the elementary school, staff development, curriculiom develop- 
:nent. and evaluation and pupil appraisal components. Each of 
the five projects we reviewed is discussed belov/. 

Project A 

This project was implemented during 1971, the first 
year of the program, and Federal funding support will end in 
June 1976. Total Fec-eral funding over the project's 5-year 
operational life will be abou; $6.5 million. 



Project A identified three issue.^ as its central 



theme : 



Prevalence of institutional racism in the educational 
system. 

Lack of cultural pluralism in the schools. 

— Need to institute educational reform. 

The project initially implemented 21 alternative 
approaches to education in cl.=^sses ranqing fxom kinder- 
garten thiough 12th grade. During 1974 ,severi:>l less promising 
alternatives were either terminated or merged so that the 
project "s full resources could be devoted to those alterna- 
tives showi-g the greatest potential for success. As a ■ 
result, t^e project is now testing 1*7 alternatives. 

These alternatives are used to e^ucat^ about 5,ono 
of the school districts 15,000 students, with expc-riments 
in such areas as: - 

Free school governance (curriculum developed by 
students and teachers). 

The open classroom (flexible curriculum, schedule, 
and age grouping) . 

— Bilingual teaching. 

— Multicultural classroom focus. 
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— Combining paid employment of students v/ith schooling- 
— Remedial teaching. 



— Team teaching. 
Project B 

This project was also implemented during th:i first year 
of the proyram, with Federal funding to end in Mfiy 1976. 
Federal funds allocated to the project over its 5-year life 
will total about $4.8 million. 

The project's main purpose is to provide an environ- 
ment that will optimize learning for each student. Because 
the school system had a high student-tnrnover rate, the 
project has emphasised diagnosi.-.g each student's educational 
needs and devising an academic prescription to meet them. 
To make schooling more responsive, such innovations as 
variations in the length of the school day, an extended 
school year which allows students some flexibility in 
attendance patterns, job experience as a ]iin,i school gradua- 
tion requirement, and development of coeducational sports 
were tested. 



The project was initially implemented at 6 of the 13 
schools in the school distri::t and was expanded to include 
all 13 schools for the 1973-74 school year. The school 
district serves about 3/500 students. 

Project C 

This project was also implemented during the first 
year of the program, with Federal support to end in Jvme 1976. 
Federal funds provided over the 5-year life of the projeut 
will total about $6.6 million. This project's mair purpose 
xs to offer choices within the public school setting which 
recognise the individual differences among all involved in 
the educational process . 

The project h^s implemented four alternatives on the 
elementary school level: 

1. Contemporary school — incorporates new techniques 
but does not deviate greatly from a traditional 
teacher-directed, structured curriculum and school 
organization by grade level. 

2. ^Continuous progress primary and continuous progress 

.intermediate — based on the premise that each child 
..earns best by wo--king at his or her own pace. 
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Instruction is based on. a carefully sequenced 
curriculum in basic skills. Students progress 
through the curriculum without regard to grade 
level . 

3. Open school — based on tl.e assumpf ' i 
children plan their own activit ich 
and carefully organized environ le only 
learn basic skills but al£;o lea ,r more 
initiative in their own education und to enjoy 
1-^arning more than their cc .nterparts in traditional 
lichocls. 

4. F>3e school — offers a curriculum developed by 
teachers and students . This is the project's most 
experimental instructional pattern and is limited 

to a small number of students and teachers.. Student 
selection of curriculum and development of a pos-'tive 
self-concept are emphasized. This option is avail- 
able to studcmts at all grade levels. 

At the secondary schod level, this project offers a 
wide range of curriculum aiternatives . Students, with ^-heir 
PrJ^J^L ^""^ allowed to design their own educational 

graduation programs must meet criteria established for 

About 2,200 of the school district's 67,000 students 
participated in this project. 

Project D 

This project was implemented during the second year of 
the program with Federal support to end in June 1977. 
Fecleral funds provided over the 5-year life of the project 
t total an estimated $4.4 million. The student enroll- 
^ericfn district is predominantly Mexican 

The project's main purposs is to maximize the intellec- 
SrLrnr-o°''^^\P°^^J?^^^^ °^ students by changing tht educa- 
^.^^r^i.rn^''^ '"^'"f compatible with their experi^^nces , 
^'i^J'owi- fu^^^'''-^"'^ personal characteristics . To accoiP.- 
plish this the project developea a progrr-M v,-hi.ch: 

— Reflects the students' cultural, language, and 
economic characteristics and is compatible with 
their learning style. 



— Enables 



students to progress according to their ability. 
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— Improves student achievement in basic; skills. 

— I? process^crionwed and geared to individual expres- 
sion, appreciation, and achievement. j 

— Promotes' social and interpersonal growth. 

— Actively involves students, parents, and cott' 
in its development and i"\plementation. | 

According to an NIE official, the project has 3hi ad 
its emphasis from a comprehensive approach to education to 
a bilingual^ multicultural education. The project now 
provides initial instruction in the students' dominant ■ 
language. ' 

About 5,000 of the school district's 23,000 students 
participate. 

Project E 

This project v;a3 also in^lement.ed during the second 
year of the program, with Federal support to end in oune 1977. 
Federal funds provided over the 5--year life of th'T^ project 
will total an estiraated $6.1 million. About 4,5 00 of the 
school district's 57,000 students participate in the project. 

The main purpose of this project is to meet the educa- 
tional needs of participating children by individualizing 
the educational process. The project is attempting to change 
a traditional school operation to one which emphasizes the 
practical use of basic skills, occupational preparation, 
attitudinal education focusing on the expressive arts, value 
foriiiation, and creativity. 

This project has begun implementing a plan which will 
allow other schools in the school district to become familiar 
with and then adopt both its proces s of change and some of 
its products. 

SCOPE OF REVIEW . 

Our review was conducted at HEW and NIE headquarters in 
Washington, D.C, We visited pirojects and their onsite 
evaluators in --California, Minnesota, South Carolina, Texas, 
and Washington. We interviewed HEW and NIE officials and 
reviewed policies, -egiulations , procedures, ^ind practices 
for administering the Experimental Schools program. 

At projects, we interviewed officialL and examined 
proposals, plans, correspondence, records, and reports. We 
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also interviev/ed officials of contractors hired to evaluate 
the projects and examined their contract files, plans, records, 
and reports. We reviewed consultants' reports prepared on 
both the projects and the contractors. 
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CHAPTER . 2 

OPPORTUNITIES T O IM PROVE PROGRAM MANAGEMENT 

The Experi.Tiental Schools program will provide information 
to educators and researchers on how to plan for and to 
implement comprehensive educational changes and the effect 
o^^ such changes on students, teachers, administrators/ and the 
community. The information will be limited, however, because 
the program was not adequately planned and implemented. For 
example : 

Projo<-' plans which could not ^ used to 

effe nt and evaluate compi hensive 

educatxwiMX changes . 

— Project evaluations were not adequately planned or 
implemented in a timely manner. 

— Neither adequate baseline data nor effective 
comparison groups were established. 

Only linr* ':ed cost data was accumulated at some proj- 
ects. This data is important to other school systems 
considering similar projects and to researchers study- 
ing the cost of implementing comprehensive changes. 

— Neither the program nor individual project: estab- 
lished clr-ar/ measurable objectives in te' of 
outpur or impact. 

To avcd-d ^ese problent^ in f uture educatione research 
and developiSBx programs, NXE ^oul d ass^ure that - research 
questions tc :rs answered are clearly identified aii thrat 
evaluation i properly planned: :and implemented. 

INADEQUATE T ROJE CT P_I'ANS 

The five projects we visited generally had net: prepared 
plans which' could be used to effectively implement and 
evaluate comprehensive educational changes. The plr:ins were 
vague and were written in conceptual rather than operational 
terms - 

Progrram instructions provrrsed to school distr cts which 
received z:.laxi=:s\q grant: requirrrrnd that plans (1) d 'fine the 
problesns iisr solved, (2) sta^ the goals and or~ ?tr^es of 
the prrjpasHd: J^ssearch project, 13) describe the 3C h^^fe to 
be use^ ^ rs=frieve the stated :^als and objective::. and (4) 
special' a is2sriir)d of document inrc: and evaluating th-- project's 
succes.. rosroe^er, the projecir plans accepted by L..e program 
did not fulSf-1-1 these reqmremEEnts . 

10 
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Inadequara documenta tion 
of educational p rob lpms 

The Experimental Schools program was ' initiated on the 
basis that varying and identifiable proble:ns exist within / 
current educational systems. Projects selected for partici- 
pation were to try to correct the problems identified within 
their respective educational systems. 

HEW rules and regulations for the Experimental Schools 
program required that project applications set out project 
goals, including (1) the kinds and purposes of learning f 
experiences to be provided and (2) the educational problems 
to be addressed— that is, specific problems of students in 
the target population. , 

I 

Projects, however, v , re not required to assess educa- 
tional needs to document the specific problems to be salv-d 
Project officials and officials of contractors hired to 
evaluate the projects told us that the problems set out in 
project plans v.-ere generally based on educational needs as 
perceived by school district officials, parents, teachers, 
or project officials. Little or no data was collected to 
document these problems. 

For example -le ~2^_ fox project A identified three 
problems in the ; ^ol liLEtrict: 

— l-he prevalj— 3 cet institutional raciRTn iri the educa- 
tional sysT3ir:_ 

— The lack of -Iz.: irsl pluralism. 

The need to — : i---te educational reform. 

Accrrding to -.t-=._j5e., these problems resulted i" n 
uiiequal education: 'ss^.^s. The plan did not define these 
problems in terms -:::_=r students were experiencing them 
"f ^^^^^.^e^'^f'-ti' .^:r, terias and concepts such as 

A^An.Tl^°''^\ "oultaral pluralism" were not 

fSoH I ^^""^ -^^^nt of consistent and measurable 
methods for correrr me problems. 

Officials at - ?-roject told u3 that needs were not 
assessed and the • jz^s cited were not documenxsd. They 
said they were no re^nzr 3d to assess needs and -r^e needs 

K^" ^^^^f ^ ^^^^ ^^'^ "^^^^ °f the s^dents as 
perce.Lved by school -rnsrr-ict officials. 

e^H^^r^ educational :trrn::_lem cited by project C was that 
schools were not meete students' individual needs. To 
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correct this, the project established a system of alternatives 
to the traditional educational structure . However r the studen 
needs which allegedly were .not being met were not clearly 
defined. According to project officials, the selection of 
alternative schools as the project strategy was based on 
community sentiments and parents ' perceptions of sturJents * 
educational needs. The officials said students* neer^.j. were 
not assessed in formulating alternatives. 

In project B, the plan was not based oxi the problems of 
students, teachers, or others in the school district; 
rather, it was based on problems identified during a survey 
made by a State agency of a neighboring school district. 
The project director told us he selected from the survey 
data those needs he believea applied to his school district. 
The statement of perceived needs was then present«=»d to a 
group of parents, teachers, and scudents in the school 
district for their input and concurrence. 

Implementation o r oblems 

Four of th ' five projects had problems implementing 
comprehensive educ^.tional chanae^.. Officials at some 
projects blamed this on vagu(- plari'i. For example, because 
the program approved an inadequat-.^ project plan^ project D 
had great difficulty implementing the desired changes during 
i ti.", first 2 years . 

At this project: 

— The components had been implemented in a piecemeal 
and fragmented manner. 

— Components had been designed ior and directed to only 
a limited nxamber of participating students. 

— Many project activities had been designed by indivi- 
dual project components rather than by a coordinated 
effort of all components. 

Project B experienced problems in implementing clianges 
because, according to one project official, its approved 
plan was not specific. We found it wr.s more conceptual 
than operational and could net readily be used ss a ba^is 
for implementing the project within participating scliools. 

All official at this project told us that imp:'.emen±ation 
was left to the principals and teachers of the si:: partici- 
pating schools. The principals told us that, because the . 
plan wa3 vague, implementation decisions had to be made with 
little guidance. 

12 
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The project lacked consistency since eacl. participating 
school developed its own imp! sinentation methods with little 
consideration of -/hat othor schools were doing. Program 
officials became aware of this problem and toward the end of 
the project's first year infoxmed project officials that: 

"Even with the components implemented in the 
project, the project as a whole lacks unity, consis- 
tency and articulation.. Each school appears to be 
responsible for (and allowed to pursue) its singular 
course without coordination among and between schools, 
teaching and administrative staffs. Unles . there is 
some intervention in terms of planning and directing 
the project which supercedes the building level, 
there is grave danger of continued f ragmentat j on 
of the project." 

Subsequently, the school district established a 
committee of school principals to solve this problem. 

inadequate plaicting and 
implehe;>itation of evaluations 

Experimental Schoo.Is program project evaluations are 
important to educators, educational researchers, and others 
obtain objective information on the projects 
Program realized this importance, the various ' 
levels of evaljiation originally scheduled were not planned 
and implemented in a mann-r which could provide adequate 
information on the impact the projects on students 
teachers, admiinistrators , s~zd communities. 

tho nVl"^""^ ^t^^^^^ levels ^ evaluation were planned by 
the Office of Education. E^h project is required to U) 
have Its own e^luation comnent, which can help modify and 
improve me project on a cc=rrtinuing basis, and ^2) assess its 
as lei^r?^ °^ objectives. This evalukion., knovm 

as level I, is funded as an integral part of the projict 
Each project is required to develop an evaluation plan ' 
acceptable to the program. ^ 

To -.^x? secoi-d level of project evaluation, known as 
tf/ti I' performed by independent contractors selected 
by the Experimental Schools program. These evaluatio^con- 
tractors are physically located at each project site 5hev 
report dixectly to the program officials^nd .re fel^^nsilL 



— Mea=..uring students' progress iia meeting project 
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— Identifying and docuinenting successful and unsuccess- 
ful comprehensive educational practices, paying 
attention co the replicability of such practices. 

— Describing and analyzing the forces which influence 
the project. / 



Systematically documenting project activities. 



— Identifying successful evaluation and documentation 
concepts, methodologies, and instrunr-'.^.g , 

A primary objective of level ii evaluation is to provide 
^-•j, 3 --sr-tfe -fact explanation of what occurred and why 
and U,. tralue judgments of what occurred. NIE plans *-o ' 

^^^^f'^V^^? ^^t resmilits to practicing educators, 

particularly those in oth.^r sctools serving similar areas; 
educational researchers, especially those interested in 
strategic pertaining l:o comprehensive educational changes- 
norn^H w^lT^^^-^ Federal, state, and local levels con- ' 
cerned wo^th improving education. Level II evaluation plans 
must also be approved by the program. ^ 

^,r.i,7?^ Office of EQucat:i .. also antirnpated funding an 
evaluation of all aspects ..;f the prcqn.v.- ,• however, this 
^nown^:'"''? f ""^^ ^^^"^ implc-meTTted as initially envisioned. 
Known as ^evel III, it will i.o funded directly by the pro- 
InrT.-H.^'^/r^"- . ^v^..luat.on was initially ex^ected^^S 
include a detenninaticn of the z-plicability of activities 
at each project. In addition it was to assess each level Ii 
pSgrlm'Ss^Jf.''^" ^"^^^^ °^ the Experimental Schools 

^"•^ request for proposals ~or level IXI evaluation 
asuec? '""^ ^evis;ed the objectives for this 

aspect of the program. According to the request, level 
issuel cross-site analyses of selected educational 

issues. ,hese issues are to be grouped into two cr.tegories: 

— issues involving school programs which hava common 
themes, such as options/aLternatives or individual 
instruction. 

""ff^yu"^ relating to implementation of programs, such 
as the role cf the Federal Government, involvement 
ox teachers and other staff, and use of level I 
evaluation data in planning. 

f.^^^^^r--' evaluation is made up of two phases. The 

^-^'^ists of planning activities to identify 
specific issues and identification of the availability of 
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data for analyzing these issues. The second phase will 
consist of analyzi-ig those issues. As of Septeniber 19 75 
the first phase was underway and NIE planned to issue a 
request for proposals for the second phase by Decembe -"5. 

Problems with level I evaluation 

Level I evaluators have had Ix.sii ted success in improving 
pro^Gcts and assessing their results. Level I evaluators at 
three projects we visited did not clearly understand their 
purpose and operated for long periods of time without approved 
evaluation plans or with approved but inadequate plans. 

For examplii, at project D no level I evaluation had been 
implemented during the project's first year. The director 
of the level I evaluation staff was not' hired until the 
beginning of the project's second year. The level II 
evaluator at project D commented on this situation in a March 
1974 report to r.IE: 

"Although the pjfoject has been in operation 
since mid-summer of 1972, Level I remained com- 
pletely unstaffed until September 1973. It appears 
that the absence of a Level I function during^ the 
first year of [the Project] was related to an 
apparent conflict between [the School District] and 
[the £:xperir:.ental Schools program] over programmatic 
aspects such as the Level I evaluation's autonomy, 
and policy differences concerning both salary and 
criteria for the recruitment of a Levul I Director. 
The failure to hire a Level I Director or otherwise 
initi-te the Level I function appears to h^ve been 
a significant factor in the failure to implement 
an effective * * * program for over a year of the 
project's existence.. To date, the Level I Director 
remains the only staff person in the [Level I] 
component. " 

Experimental Schools program officials told us that the 
1-year delay in implementing the level I -valuation occurred ■ 
oecause z 

— Ti.- project's director did not want to .lire a level I 
director who would earn a salary higher than his 



own. 



-The project's director wanted to completely control 
all level I efforts. 

-A misunderstanding existed between program ann project 
officials as to the qualifications for the leve^i I 
director. 
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Generally the p. rot give projecr ilod 

guidance for preparing . laation plans. at project 

E the level I evaluation ope :at . for about 2 years before 
guidance was received from the Exj^ erimental Schools program 
on how to develop an evaluation plan. When project EVs 
plan was submitted for approval, program officials considered 
It unacceptable because it did not include the specific 
evaluation questions being addressed, specific program areas 
to be examined, or the rationale for such examination?; • The 
project's l€ /e:. I evaluation plan was resubmitted and finally 
approved about 3 years after the project was fund'.d. 

The level I evaluation plan for project A, though 
approved by program officials, was vague and generally did 
ncpt consider the project '5 goals and objectives. It con- 
sisted of three elements: 

— Observations by level I staff. 

— Interviews of samples cf students, parents, teachers, 
and administrators. 

— Standardized tests of basic skills accomplishmant 
given to students. 

The e^ra] uation plan stressed the importance of observc-r i on 
but made little reference to how such observation would 
lead to analysis of project effectiveness. in addition, 
the plan did not specify how the information gathered 
through observations, interviews, and tests would be used 
or what kinds of reports woulc* be prepared for project 
managers. ' 

According to project A officials, after 3 years'^of 
operation, level I evaluation had little success in pro- 
viding information, on projcot 'effectiveness or in helping 
to develop effective methods for achieving the project's 
goals and objectives. One official told us that level I: 

Had neither presented any usej.ul products nor been 
instrumental in any project decisions or changes. 

— Was inconsistent and ineffective and lacked the 
direction v/hich could have been provided by an 
adequate plan . 

Pr ob^lems with level II evaluation s 

Each level II evaluation contractor had difficulty 
developing acceprable evaluation plans. As a result, 
the levBl II evaluations will not provide information 
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on the impact of implementing comprehensive educational 
changes from the inception of the project through its 
5-year life. Level II contracts at projectn A and E 
have been terminated because of the contractors' inability 
to prepare evaluation plans acceptable to the program i 
after about half of the projects ' operating lives had i 
expired. it took the three other level II evaluators /- 
between 12 and 22 months after the award of their -on- .* 
tracts to develop acceptable evaluation plans. In our 
opinion, the program's failure to identify acceptability ' 
criteria for evaluation plans was a primary cause of ' 
problems in developing thf*se plans. ■ I 

i 

The Office of Education originally estimated that 1 
developing an acceptable evaluation design would take I 
from 3 to 5 months after contract award. In its initial ' 
announcement. o^ a competition for the Experimental Schools 
program, the Office stated: 

"A ^-econd lev-1 of evaluation will be designed 
and implemented by the Office of Education in 
coordination and conjunction with each experimental 
school project in order to assess the strengths and 
weaknesses of the project's comprehensiveness. in 
audition a single evaluative design will be developed 
by the Office of Education in oi^der to insure that 
comnion instrnments will be used to assess replication, 
treursportability , and comparable data among the 
experimental school sites." 

The Experimental Schools program, however, required 
each level II evaluator to develop its own evaluation design 
Level II evaluators at projects funded during the proqram's 
first year were to: ^ tr ^ ^ 

— Identify ^.rxd explain the measurable operational 
goa:ls and objectives in (but not limited to) the 
areas of: 

1- Student achievement and attitudes. 

2. Staff development and performance. 

i- Community accepLance and involvement. 

4- Overall project development 

Prepare a plan to measure .progress toward and 
acconiplishment of these operational goals and 
objectives. 
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— Prepare- a plan for systematically identifying and 
observing relevant forces which might influence 
the overall project. 

— Establish the requirement:: for an information, or 
data, system to meet the needs of levels I and II. 

— Formulate a report format setting standard? for 
the content and frequency of reporting and for 
documentation, and 'determine the rellrtionships 
among the various activities requi;ri'ng documentation. 

— Identify and analyze costs incurred in implementing 
(1) the project as a whole and (2) individual project 
components. 

The requests for proposals for level 11 evaluation of 
the two projects funded during the second year of the 
program contained similar provisions. 

Each of the two level II evaluators whose contracts 
were terminated submitted four evaluation plans or revisions 
for approval by Experimental Schools program officials. Ncne 
were approved. 

The contract for project A's level II evaluation v;as 
awarded on June 30 > 1971. It was for about $750,000 and 
was to conclude on December 31, 19 73. At that time the 
evaluator was to be considered for refxanding for the 
second 30-TOnth period of tJie project. 

In October 1971 the contract was transferred to another 
firm because of logistics problems experienced by the 
original contractor. The onsite evaluation team, however, * 
remained intact . 

In January 1973. a team of NIE consultants reviewed 
this evaluator* s work and reported to NIE that virtually 
no research products had been produced. These consultan ,.s 
reviewed the level II evaluator's work as part of an 
assessment of all programs transferred to NIE from other 
Government agencies. They reported that the onsite team 
had not been able to develop an operational plan that v;as 
acceptable to program officials. In January 1973 all mem^ 
bers of the onsite level II evaluation team either resigned 
or vrere fired by the evaluator. NIE subsequently terminated 
the contract for level II evaluation at project A. 

In April 1973 NIE issued a request for pioposals for 
level II evaluation at project A and the terminated con- 
tractor v/as again selected as the level II evaluator. 




According to a program official, this contractor was 
selected because its proposal was ranked highest of all 
those received and also had the lowest proposed budget- 
The contract was awarded for $846,413 for the period 
June 3-3^, 1973, to September 30, 1976. The level II ' 
evaluator developed an evaluation design which: i 

Stated project goals in measurable terms. I 

— Identified tests which v/ould be v<5cd to measure 
the project's effectiveness. I 

I 

Established specific milestones for delivering I 
reports to NIE. 

— Identified necessary baaeiine data and methods for 
generating it. 

— Allowed tracking of individual students. 

Established procedurefj for coitiparing the project's 
effectiveness with that of the entire school district 

The level ll evaluator told us, however, that the original 
onsite staff had not developed any data which could be 
used m the r-.irrent evaluation of the project. As a 
result, the project A evaluation will last only 3 years 
instead of the 5 originally planned. 

The level II evaluation contract at project E wa<? 
also terminated. The contractor cited the lack of 
measurable project objectives as being the primary reason 
for Its inability to develop a satisfactory evaluation 
Plan. NIE terminated the level il contract in Sep^eS£er 
1974, over 2 years after the initial award. 

^ September 19 74 letter to NIE, we questioned the 
continuance of project E without a level ii evaluation 
because, m our opinion, a project should not be funded 
by T±e Experimental Schools program without adequate 
evaluation. ^ 

In March 1975, however, NIE issued a request for 
proposals for level II evaluation at project E. NIE stated 

^ "The request for proposals is consistent 
with an NIE review of the project which concluded 
that an impact assessment based on the presumption 
of comprehensive change was unrealist.-' c and would 
be replaced by a documentation and analysis of 
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the new instructional model at the project and the 
factors favoring and inhibiting its implementation. " 

_ The request for proposals stated that, although the 
project had no level II evaluation from August 1974 to 
June 1975, data for this period would fae collected by 
the pro:ect. Further, data collected by the former level 
II evaluator was to t>e available to the new contractor. 

In our opinion, because of the problems experienced 
at aii levels of evaluation, nie has lost the opportunity 
to evaluate whether comprehensive educational changes can 
greatly and permanently improve education at project E. 

.^.^^^^1 II evaluators at the other three projects we 
Sr Derio^''^ not able to develop acceptable evaluation plans 
ror periods ranging from 12 to 22 months after co-ntract 
award. According to two of these evaluators, a primary 
reason for their inability to develop such plans was a lack 
ox guidelines from Experimental School? program officials.' 

example, at project d the contract for level ii 
evaluation was awarded m June 1972-. The evaluation nian 
was. not approved until 12 months later. Officials of' 
the level n evaluator told us they had only the request 
ror proposals and inconclu.-.ive discussions with program 

^^fi*^!^ "^^"^^ ^^^^ the evaluation plan. They 

said a continuing problem in developing data for the 
experimental Schools program has been a lack of speci fi- 
nations as to the subjects and the depth of evaluations. 

st^teS"i-h i?^""""^ ^^^^ "^^l?""^ ^° NI^' this level II evaluator 
by data cS^L^^°'' pjoblem had been time pressures generated 
pfr 1 f collection and revisions of the evaluation plan The 

Level III ^evaluation 

level'^II^f^^r °J ^^'tnf.^^°'' °figin^-^-y anticipated that the 
evei 1X1 evaluation would include an evaluation of (1) =11 

KS^o?!'^ ^''''^^ evaluations and (2) the Experimental 

^rt d%r°^f^^^^^ a whole. However, it has not been imple- 
f^^^^ho ? anticipated. In Apr.U 1974 NIE requested proposals ' 
de^.'deS i^r^^ evaluation. m June 1974, however, ?t 

decided not to- award the contract at that time because: 

—Both the projects and the. level II evaluators had 
been reluctant to provide data to the level III 
evaluator. - 
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— Prograjn officials did not have adequate knowledge 
about the extent and character of data at each 
project. 

— It was unsure of the cost of analyzing and 
processing the data for cross-site comparison. 

As discussed previously, NIE has revised the objectives 
for level m evaluation. 

ADEQUATE BASELINE DATA OR I IF FECTIVE 
COMPARISON GROUPS NOT DEVELOPED 

None of the projects we visited accumulated adequate' 
baseline data on student achievement and attitudinal levels 
either before the comprehensive changes were institvi-ed or 
early m the operation of the projects. As a result evaluators 
will not be able to fully determine the impact of the com- 
prehensive educational changes implemented over the entire 
5-year project life. 

Although achievement tests were given to project B 
students, the level II evaluator " told us that the data 
accumulated was narrow in scope and not necessarily 
representative of the eiit:ire school system. 

iAt project C, the school district gaye basic skills 
tests to all students and attitudinal tests to secondary 
school students before the project was implemented. However, 
project officials viewed these tests as inadequate for 
determining project impact because they did not address 
the specific attitudinal factors with which the project 
was concerned. Level Ii officials stated this test data 
tended to have a variety of problems which detracted from 
Its validity. 

Also, groups of nonproject students which could be 
effectively compared with groups of project students were 
generally not established. However, leveJ II evaluators 
at two projects did establish comparison groups. At projert 
D, the evaluator accumulated achievement data for both croups. 
At project A, data on these groups was not accumulated until 
the start of the project's third year. As a result, an 
effective assessment of the impact of comprehensive changes ■ 
over a 5--year period will not be possible. Data was not 
gathered on a timely basis at this project because of problems 
with the original level II contact which was terminated.. 
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NIE officials told us: 



— Reseai.chers encounter problems in establishing / 
compaxison groups in a real-life setting such / 
as the Experimental Schools program. 

'/ 

— One difficulty is experimentally selecting and 
controlling both participating and nonpartici- ; 
pating students since it is almost impossible to' 
prevent a new practice or technique from being, j 
used in nonparticipating schools. 

i 

— When comparison groups cannot be effectively i 
implemented, complete baseline data on project , 
participants should be accumulated to permit some 
measure of the program's impact. 

NEED FOR COST ANALYSIS 

According to NIE, cost dala is an important factor 
in implementing such an educational innovation as the 
Experimentr-i 1 Schools program. Guidelines provided to 
the level ij. evaluators require that a cost analysis be 
performed and that the level II evaluation plan include 
procedures for such analysis in the areas of 

— component parts oZ projects, 

— incremental versus operational costs , and 

— phaseout of Federal funding. 

At four of the five projects, however, no cost analysis 
was being performed by level II evaluators at the time of 
our visit. According to NIE, understanding (1) the cost 
of an innovation and (2) shifts in the traditional spending 
patterns to acconmodate the innovation are important to : 

— The educational practitioner trying to decide on 
utility, adaption, an<3 iinplemehtation- 

— The educational researcher trying to understand 
the process and problems of phasing out Federal 
funding. 

The level II evaluation plan was to include procedures 
for identifying the costs of implementing the project as . 
a whole and of the individual project components. The 
analysis of incremental costs was to include the project's 
use of Experimental Schools program " funds and an estimate of 
probable costs of implementing project components in other 
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school systems. These costs were to be expressed in terms 
of the need for additional staff having particular abilities, 
release time, and training rather than in terms of dollars 
because Of the nationwide differences in personnel salaries, 
in addition, the level II evaluators were to assess and 
document the progress of school systems in reducing project 
costs so the comprehensive changes could continue 
after Federal funding was discontinued. 

Only the level II evaluator for project D h:.d performed 
any type of cost analysis. This analysis was aimed at 
demonstrating 

—how incremental funds provided by the Experimental 
i>chools program were used by the project, 

—how the declining real-dollar budget of the school 
district was related to expenditure decisions, 
and 

how the projc'^t handled the phaseout of Federal 
funds . 

^ ■'"f''^-'- evaluator told us, however, that ^ise 

of the cost analysis will be limited in measuring :the 
impact of expenditures on students. 

At two projects, level li evaluators told us they 
planned to initiate some form of cos- analvsis at a 
later date. The reasons for their failure' to make cost 
analyses varied. At project A, the evaluator told ue the 
^ni^-^r^^"* refused to allow it access to the necessary 
financial records. After notifying NIE of this problem, 

Sft^Satr''°^>,"^^ ^^^^ ""'^^"^ generate the necessary 

cost data. The school system, however, also refused NIE 
access to its financial records, even though such access 
oro-^ejrj S ^? project's grant agreement. 

^n^lvS. ho l^^'^l.^^ evaluator did not perform a cost 
analysis because this was done by the level I evaluator. 



NEED TO ECTABLI SH SPECIFIC 
OBJECTIVES "■ 



The Experimental Schools program did not set out 
^^^fnfee' "^f f ^'-^^^blG objectives for evaluating its effec- 
tiveness. Also, individual projects were not required to 
^^^W?.-f/"'^^^'' objectives, which would have allowed for 
^??Hn "measurement of the effectiveness of a program 

involving comprehensive changes. fiuyram 
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Program objectives not specific 



HEW's operational planning system re^ruires aget: 
program manaaers to develon ^^^;^r measur^^iDle objecrr:^:^ in 
terms of o; or impact : -i et milestones to s^s^are 

effect.vene The system i haxicr^ook says that ircsmcrt; objec- 

tives state terms of how ressnnrces and output ^^^1 affect 

a problem . ::^ed are preferable to output objecxr^es stated 
in terizs o. rme activities or products expected f r - a 
certain levj;_ resou^-'ces. The Experimental Schc program 
has not est-J±)l:i5hed such specific objectives. 

As staraj: in the planning system document for f ^^scal 
year 1971 — Th^rr program's first year — the program' s r?jective 
was to select: from four to six projects and to fun cheiti at 
a total cost czf about $15 million. At least one project 
was to be in ^ urban location. Another objective was .to' 
provide techixical assistance to each project for finalizing 
plans and developing specific program evaluation criteria. 

In fiscal year 1972 the program's objective was to 
monitor the develo|>ment of three fiscal year 1971 operational 
'Sites, to initiate operations at chree to five new sitas^ 
and to implement an evaluation and documentation plan. 

Since the program was transferred to NIE, no new 
operational planning system objectives have been set out 
for it.- According to an NIE official^ initial effort.s 
were aimed at establishing agencywide objectives and a 
framework for an NIE management system rather than at 
establishing objectives for individual programs. 

Project goals and objectives 
not specific 

Projects generally did not state their goals and 
objectives in specific measurable terms; rather^ they stated 
them in vague ^ conceptual terms which made -difficult for 
evaluators to assess projects' effectiveness. Generally 
the five projects did not specify (1) when their objectives 
would be met, (-2) interim milestones which could be used to 
monitor progress, or (3) the devices to be used to measure 
how well they met their objectives. 

For example, project B selected as its main goal 
providing each student an optimum environment for learning. 
One objective established by the project to reach this goal 
was to decrease by 75 percent the number of students 
behind in grade level. The plan for the first 30 months 
of the project, however, did not state tlie academic subjects 
to be. monitored or when the objective would be met. Also, 
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•-^-m iniles±cr.es or the - ::r:.y to be 
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n 



^vhlxc S: Tj^ — must be even more responsive to 
the indiv. u a^s served and must do more to 
stimulates individual growth and fulfillment. " 



-"Choice making by students-, teachers, an 
will become the basic way of school, life 
members of [the Project] prograitc within 
public school sector * * *, poj- t^uaent 
goal of the program is to encourage thei 
growth arid development of positive self- 
by allov/ing them more self-determination 
giving them skills and guidance they nee 
the increased responsibility. " 



d parents 

for all 
the 

s a major 
r personal 
concept 
and by 
d to handle 



— "While a major emphasis of the project is on the 
affective domain it is also obvious that program 
participants are expected to be at least as 
successful as other students in the school system 
m the area of cognitive development." 

Because of the way in which the objectives are stated, 
evaluators will not be able to measure the project's 
effectiveness. 

Project E's objectives were aJso stated in general terms 
and could not be meas^ired. Following are examples: 

~"To parovide - -riences for students and teachers 
design:ed t3D p rroote positive attitudes towards 
self, learsiisc arid positive relationships with 
others." 
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— '^r^jD provide the means for each student who .ta5'^ 
nisstered the basiLC skills to design h.is owr 
edxiccitionel procxam according to his needs an 
value structure." 

CONCLUSIONS 

The objective of the Experimental Schools procr eu" is 
to test the hypothesis that comprehensive educatior^^ 
changes will result in improvements in the way studf -s 
are educated. The program will provide informatior the 
process of change within each school district, inciu:„ng 
how changes occur and what factors impede or facilirr^na 
change. 

However, the program's ability to provide information 
on the impact of comprehensive educational changes on 
students, teachers, the school system, and the commtm- ty 
to educators, educational researchers, and other interested 
parties will be limited because (1) inadequate project plans 
were-approved by the prograra, (2) level" I and II evaluations 
were inadequately planned and implemented, (3) important 
cost analyses had not bee- performed, and (4) neither the 
program nor individual projects developed specific objectives 
in terms of output or impact. 

Although the Experimental Schools program is scheduled 
to be completed in fiscal year 1978, important lessons can 
be learned from the problems it has encountered in planning, ' 
implementation, and evaluation. T^.ece lessons can he applied 
to other programs, especially those rn educational research 
and aeyelopment. 

RECOMMENDATIONS TO THE SECRETARY OF HEW 

We recommend that the Secretary require the Director 
ot NIE to take the necessary steps in future educational 
research and development programs, wherever applicable, 
to insure that: 

Project plans approved for funding include (1) 
docuuBBntation of the need for the educational 
changes sought through the research, {z) specific 
mstractions on how to conduct the research, and 
(3) specific, measurable objectives in terms of 
putpmr or impact. 

Evaluations are planned and implemented so that 
they provide necessary impact and cost informaticao 
over the entire life of the project. 
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hQKllZ CClMMEN^ AND OUR E^.TiLl QN ' 

In a letter dated February 23, 19 ""6, . (iiree app. I) , 
HEW coiUiTi^nted nn matters discussed in rhis report. Althrr gri 
HEV7 gener-ally ::zoncurrec with car reccmssendations , it e^iprassed 
several -Dncerr:s. 

HEW agre^ that the Director of EIE should, wherever 
applicable, assure that project plans approved for funding 
include xnose items cited in our recoomeiikdations . HEW 
believes^ hcwever, that curreixit NIE regulations and direc- 
tives which require rigorous rsview of pmject applications 
implemeni: our recommenaations ^ 

We agree that these regulations and direct ix^es can,- in 
the future, help prevent the types of management problems 
identified. However, the Experimental Schools program was 
not subjected to the rigorous review called, for by the 
regulations and directive's; otherv/ise, many of the problems 
cited in this report could have been identified and corrected. 

In an April 1975 report to the Acting Direc-:or of NIE, 
an internal committee established to review the Eiiperimental 
Schcols program staiied that NIE had failed to rigorously 
review the program before that time. The review committee 
also said the level of expenditure, evaluation methodology, 
and other program characteristics were all products of NIE 
management and policy decisions. 

HEW also commented tnat our report did not contain 
sufficient background on large-scale social science research 
and development methodology between 1970 and 1975 or the 
difficulties associated with such research and development. 
We do not believe thar such a discussion is warranted. Our 
report assesses the management of the research and develop- 
ment performed. We do not believe it ±3 appropriate to deal 
with program management zin relation to tthe state of the art 
of educational research and development 

HEW also qisstioned riie number of projects we reviev/ed 
and the timing o± our revn^ew. We reviewed the first five 
projects &inded. after thev had been in sistence for 2 to 3 
years. Wh±lp^ agreeing w^h our assessm^t of problems found 
at these five sites, IffiW r-elisves that, because they were the 
first proiects funded^ t--ey suffered mast from state-a:z-the~ 
art probl^s. Further^ HEW s^v.-ted that a review in ths 
second ami tL^^rd years ax 5-y r proje.r:-s misses out cm what 
can yet bs l^earned from these ^rx:j$ct3. 

We d %! S c a cr ree. The five prrrjscts reviewed will reisivB 
an estimatsed S36 million, or sixornr 65 oercent, of the $55 
million tcr be spent by the Exj^rlmental Schools progrsim. 



T ey hav ~i been in existence for ne longest period of time and, 

our c pir^ian, offered the gre^:.3St opportunity to assess 
rrrogram raniagernent. While we agTr-^ee with HEW that much can 
L:. learr.~:3d frm the program, we b elieve that the program's 
Ex ility tc prrr-zide inf otTr.ation c: the impact of comprehensive 
e :.ucaticna^ ~anges will be limi 3d. / 
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CJIAPTEn 3 
COMPLIANCE WITH PEDZPAL FINAI>ICIAL 



KEQUIREMENTS NOT ASSURED 

Special financial requir ^r.er/~s for the Experimental 
Schools program are stated as fc ^ows in 45 C.F.R. 151.56: 



"Federal financial assLstance * * * may not 
exceed the difference between (1) the tiotaJl cost of 
the project and (2) the nuna^er of students in the 
project multiplied by the average per pupi2 expend- 
iture (as determined by the Commissioner) tot the 
area ti:: be served with respect to such per±C3d. " 

The E^Esrimental Schools progrsm did no-^: assure that: 
participatiiiA, school districts could provide tihe financial 
data neceFoary to determine c jmpliszice with tihese regula- 
tions. At three of the five projects we focmd that the 
records did not provide this type of data* 

The announcement of a ccT^pet±t±on for E:::q::;rimen±:zl 
Schools grants c'ated March 31, 1971, stated til; .it Federal 
support would be limited tc cremeixtal cost-: of project 
inplementation, such as co^zzi for developing staff 
necessary to operate the pmgram, acquiring materials ♦ doing 
minor remodeling, and evalii;n:ing and documenting the rrroject. 
Federal funds were not to b used to support basic pei--pupil 
expenditures in prc^ r et sci> -:'ls^ or to support major" co:i- 
struction. 

The regulations were f :;er clarified in a memoranduia 
of understanding to one of : t projects which starred tl:'.at: 

"The Experime*^t :ioals grant is for incre-- 

mental costs only.. It undesrstood that fthe 
Project] *d.ll coiiiid^ixe - ^ proTride services, raatera^ls 
and other support & the pimils in the Experimental 
Schools pxrjgram t he ^ame ra^ and per pupil cosrt: 
as currently exis.t_ oxr may sexist in the future due 
to changes :in local statte funding patterns." 

This meiTKJLLciidum of understanding was included as a 
provision of tiie Experimentdl Schools program grant to the 
project. 

According to HEW officials, the following infortnation 
would be required to assure compliance with this regulation: 
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— Definition of the area to be serve by projects. 
Because (1) projects did not initially serve all 
schools in the school district and (2) the 
participating scliools serve only specific neigh- 
borhoods, the area to be served would be the 
area served by the participating schools — not 
the entire school district. 

— Identifi cation of average per^-pupil e::penditure3 
• at participating and nonparticipa ting schools for 
the periods (a) imiBedi^tely preceding receipt and 
expenditure ol prograir fund3 and (b) ciuring which 
Federal reimbursemen-ts are to be coit^:::.-ed. 

— Determination of total project costs each 
participating school r-on Statec local and 
federal sources. 

None of the three school districts where we assessed 
compliance were able to prcx-ide complete cost data for 
indi%ridual schools . 

At project D the schoc-L a-.;i'.ript appar^^ .7 reduced 
the lev'}l of support it pro\^ldez to participajLing studeats . 
We compared payroll expendit;:-rGS for a l-montr. ns^rioc at 
both participating and nonparticipating 3chocl;^ with cam- 
parable enrollments, found that ^otz^X pai- 11 expsirdi- 
turos at both types of school ■^er^ abo^ eq-iia. , . 
program funds were provided f ici^axi^r 3tuden:zs^ 
it appeared that the school d:av::ic prrr-vride:d less non- 
Federal funds for them. 

A school district official rold us th£zr: 

— Experimental Schools p— ■ ai?: furriin?: r::^:quireinfiiir5 
are contrary ro the scbn' distrrrrt^H philosophj^ a:f 
equal educational oppa^;::rii±y for a^l students. 

— If the school district i-rcnplied wj.th Federal 
financial requirements, students participating 
in the project would have an advantage over 
nonparticipating students. The stztxool district, 
therefore; distributees to the norrsa^ticipat ing 
schools on a per-pupil basis more nonprogram 
funds than it doer, to p.articipaliiirig rBc^mals . 

— The school district -emprs tr) ecxia.l_2:e totiai 
per- pupil expendituirs-^ for iDOth paxnicipatiim am: 
nonpnrticipating sc±ids_3 * 
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NIE officials agreed that school districts generally 
do not maintain account ing systen'S which show the average 
r»er-pupil expenditure by school. They told us that average 
per-pupil expenditures are not used in determining program 
compliance ,with Federal financial regulations . They c.lso 
told us that NIE has a system fur determining the allow- / 
ability of program expenditures, which includes (1) requiring 
projects to submit quarterly expenditure reports by approved 
budget categories and (2) financial audits at the end of 
each grant period. NIE, however, has generally relied on 
the good faith of school districts to provide the same ; 
l3Vel of non-Federal financia] support to both participating 
and :*onparticipating students. j 

CONCLUSIONS I 

i 

Neither the Office of Education nor NIE insured that 
projects used accounting systems which provided financial 
data shewing whether school districts were providing less 
per-pup:.! support to students participating in the program 
than to other school district students, 

RECOWVJ -^.Vr; ^\'S TO THE SECRETARY OF HEW 

The So* retary should require the Director, NIE, to 
institute; 

— Procr-dures for requiring program offices to verify 
that recipients' accounting systems are adequate 
t:o provide the type of data necessary to assure 
complidnce with special program financial 
regulations . 

— Procedures to assure that recipients provide this 
data to program offices. 

AGENCY COMM ENTS AN D OUR EVALUATION 

HEW agreed with our recommendations and stated: 

— It is important to distinguish between (1) standc».x-d 
accounting systems necessary to support allow- 
cJoility of costs and (2) unique systems that might 
be necessary to meet special project conditions. 

— The problems cited in our report arose from an 
attempt to require school systems to produce 
financial data which traditional accounting 
procedures could not produce — namely, data on 
dollars per student." 

38 
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— It now realizes that th"s requirement was not 
particulc-iriy realistic, especially without 
technical assistance from the Federal Govern- 
ment. In the future, it will try to insure that 
any special accounting requirements levied upon 
award recipients will be accompanied by a review 
of the accounting systems necessary to support 
such special requirements. 

We recognize that the financial requirement placed on 
projects was a special requirement of the program. We' agree 
with HEW that this requirement was not realistic because 
the projects could not provide the necessary data. We 
believe that HEW's intention to assure that future award 
recipients will be able to comply witn special accounting 
requirements should, if properly impleirionted, prevent this 
problem from reoccuring in educational research and develop- 
ment programs . 
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DEPARTMENT OF HEALTH. EDUCATION-; AND WELFARE ' 

OFFICE OF THE SECRETARY 



WASHINGTON. O.C 20»l ; 



Feb. 23 1976 



Mr. Gregory J. Ahart 
Director, Manpower and 

Welfare Division 
United States General 

Accounting Office 
Washington, D.C. 20548 

Dear Mr. Ahart: 

The Secretary asked that I respond to youx request for 
our comments on your draft report entitled, '^Experimental 
SchcoU Program: Opportunities to Improve the Management 
of an Educational Research Program^', llie enclosed comments 
represent the tentative position of the Department and are 
subject to reevaluation when the. final version of this 
report is received. 

We appreciate the opportunity to comment on this draft 
report before its publication. 

Sincerely yours, 

Jchn D. Yo4ig ^ 

^sistant Secretary, Comptroller 

Enclosure 
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Comme nts of t!hpDe£^ '^ttnGnt of Health, Edivcatlon, and Welfare on the 
Comptro ller Gen erallgJg^Pgrt to the Con g ress entitled "Experimental 
ScjiooXs Program : 0^£9£i!iqjj: ie3 to Itnprovc the Mana^c^^ent of_ an 
Educational Res earc ?^ ProKram ^ Jjov^&mbe^^ 1973, B-16403l_m^ 

GAP RecongDendatipn 

jjg_ ^connnend th at tJl^,Secre^ tary of Healthy Education, and Welfare 
■Iggjjire^ th e Director* NIE^ to take the necesaary steps in futx^re 
edjjcation al researcj Lf nd^de velopiaent propfang where applicable to 
jlnsu Te that : 

— Pro; )ect pla gf^app rov ed for fuading include (1) docunentation 
of the needjfQr_the e ducational ch ange g__goug ht throuf^h the 
reBcr-rch> C^^ spacl ri c lnstruction g,j:>n^ bov to condrict the 
resea rc hj_22i^.I, speci fic ^ measurable objectives in terms 
of output ,or_J^i'yt, 

— EvaluatlonsJ g£e^ > jLanned an d lopleateuted 90 tbat th e y provide 
neceaeflxy^i^EL^. 

of the prol£ Ct. 

Depjij rtffient Corotoe nts 

We concur with the CAO recommendations that the Director should, 
•'wherever appllcabl^i"* assure that project plans approved for 
funding include tht? ^ocuioentation and information referenced In 
the recommendation- By 'Vherever applicable", we understand that 
the GAO recognizes that certain research and developfMent projects 
Buch as field initiated research grants may not be opeciflcally ained 
at educational chan?^ or susceptible to federal ^Instructlona on hov 
to conduct the research". We believe that the Institute's regulations 
and directives alrci^dy require the kind of rlgbrouft review to assure 
the documentation "wherever applicable" t »at GAO Is rec.oaaiendlng. 
(These HIE regulations and directives were not applicable to awards 
on which the GAO report is baoed since the initial awards predated 
the creation of the Institute), 

GAP R ecogRen dat ion 

We^J^ecogmcnd that tbg.^ec r^tary of Healthr Educationt and Welfare 
req ulre^ie Directo r, NIE, to tak^ rhe nec essaty steps to assure that ; 

— Procedurt;fl lQ!Lrej^iiirtnR p rogram offices to vg^ifY _.that 
grantee acc ount in(^. 8Y3te mfr are adequate to provide the • 
type o f data neceB8a jr y_ to ensure compliance %rlth financial 
regulat lons * 
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j:£_ £rogram offices. : 

Department ^pmments 

^i^ter(;fSdSta'J%."rr^"'"''°" ^""^^ procedures are appro- ' 

office; or a^^b s^ff f '""^'"'^^ contracting ; 

between stan^I,^ . stattj it is Important to distinguish 

of costs nn^ /"°"''""^ ^y"^' necessary to support allowability 
proJe^^t'coSltJo^r^ Se'^^o^Je" "^'""' 'l necessary'to .ee^ special 
arose from a special I^tinr ^ ^Pf °Prlately cited in the GAO report 
financial u ^"^'"P^ require school systems to produce 

pr^de ,-a^f '''r' ^^--^^^S Procedures did'Lt ' 

that this r^^r:^^:zst::^;r^^^^^:^ rr"- 

levSd u^of ^v^'ds wurb! J^'' ^P-cial accounting req.ire.encs ' 
accounting .^^ accompanied by federal review of the 

accounting .ysto.s necessary to support such special requirements. 

■^^Seral^^a^^nentjCoD^^ 

Solfpoit? be'i^^f ;Lrth''%:'%'''''^ recommendations in the 
sideration various hI^^n^f ^ . ''"P^" take, into con- 

SJifi'o" ^S^'Sc.'L'^' f r"'? °' background 

icvj-ew oi. xarge-scale social science R&D ttethodolopv <t 

development over the period 1970 - 1975 JJf; ^"^^'^^ 
discussion woulH r^.c^ ^„ 1 Z 7. . inclusion of such a 

Ih= difficult, of .h. .„terprUe c» b. .tttlbut.d to three factors: 
(1) The ..,™,ptto„ 1„ 1970 that the cpaclty existed at 

.Sei^rLi^^' '-i>l-«»""8 =o.l,,. . 

taSnvSualora!""'"'" "eld 
And most ioportantly, ^ 2 
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(3) The attempt in the Experimental Schools Program design 
to collapse three separate stages of social-science , 
research methodology Into a single five-year cycle: j 

(1) Designing an intervention treatment In ar.cordance 
with our best knowledge of how to achieve a care- 
fully stated and measurable objective. 

(xi) Implementing the treatment to assure the school^ 
system's capacity for consistently delivering the 
specified intervention treatment. 

i ' 

(111) Teflting the impact of a successfully Incplcjaented 

treatment on e. carefully selected student population 
against the specified objectlve(8) , 

These three technical stages are separately arduous, and to attestpt 
to perform them simultarveoualy in the same school site appears, 
from the perspective of our current under etandlng of thesa uatterft, 
tnconcclvable . 

Our second conrenn stems froru the rilsunder standing created by the 
limited nature of your lnquii7> In terti3 of t\ie number of oites 
reviewed, and the tixae selected for your inquiry. As you know, the 
Experliaental Schools Program conelsts of 18 projects, wy.lc 
agree that your assessment of the problems found at the five aitea 
you visited la substantially correct, these sltea were the flrat 
five f«jnded> and therefore suffered moot from the ctAtc-of-tha-art 
problems noted earlier. It la our view that our capacity to deal 
vlth these problems Improved over time, aa exemplified in later 
planning documents for both the school projects and contractor 
evaluations. 

Consiequently, your conduct of the Program review In the third year 
of a five year program tends to emphasize the degree to vhlch each 
of the five projects you reviewed deviate*? from the original plan, 
while omlttiuj^ what can yet be learned from what is act;uilly now 
happaniog in these and the other projects making up the program. 

Finally, In the course of our conduct of the Experlmenral Schools ' 
Program, as the consequences of the conceptual, methodological, and 
practical weaknesses we cite above surfaced, various res&edlal actiooa 
were taken. Among them are: 

The evaluation of cac'i project has been narrowed In scope 
and matched more closely to the goals and objectlvco o£ 
each achool^s project activity. 

The key ataff of the e*'aluatlon contractor in Kinne&polls 
and Bdgevood have been replaced with more experienced 
individuals. 

The evaluation contractor In Greenville was terminated, and 
replaced with another, 35 
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For each of the ten rural sites, an appropriate non- 
Participating school district has been selected for com- 
parison study. 

The Level in <cross-site study) feasibility project has 
been completed. This feasibility project was designed 
CO identify those project components that warraat compara- 
tive study across several sites, and plans for such analyses 
sre now under development. . 

Se^Zn' "^'^ accept your specific findings, but believe they 

50ciE3 c . '^"f ! description of devtlcping large-scale 

to the f^"""/^'^ methodology, and do not gi.e adequate attention 
^ene^at^ potential of the Experimental Schools Program to 

generate userul information. The National Institute of Education 

the Dir!^^"^ ^^'^"''^ ^^J^""" understand the need to assure 

de^?0D r f '^^^'^^""^'i discretion to take legitimate risks to 
aeveiop important research data. 
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APPENDIX IX APPENDIX Ji 

PRINCIPAL OFFICIALS OF THE 
DEPARTMENT OF HEALTH r EDUCATION, AND WELFARE 
RESPONSIBLE FOR ACTIVITIES DISCUSSED IN THIS REPORT 

Tenure of office 

From To 

SECRETARY OF HEALTH, EDUCATION, AND 
WELFARE: 

David Mathews Aug. 1975 Present 

Caspar W. Weinberger Feb. 1973 Aug. 1971 

Frank C. Carlucci (acting) Jan. 1973 Feb. 197: 

Elliot L.. Richardson June 1970 Jan.* 197, 

ASSISTANT SECRETARY FOR EDUCATION: 

Virginia Y. Trotter . June 1974 Present 

Charles B. Snunders , Jr. (acting) Nov. 1973 June 197 

Sidney P. Marland, Jr. Nov. 1972 Nov. 197 

DIRECTOR, NATIONAL INSTITUTE OF EDUCATION: 

Harold L. Hodgkinson July 1975 Present 

Emerson J. Elliott (acting) Oct. 1974 July 197 

Thomas Glennan Oct. 1972 Oct. 197 

COMMISSIONER OF EDUCATION (note a) : 

Sidney P. Marland, Jr. Dec. 1970 Nov. 197 

a/ Responsibility for the Experimental Schools program was 
transferred from the Office of Education to the National 
Institute of Education in August 1972. 
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